Image processing apparatus, control method thereof, recording medium

ABSTRACT

An image processing apparatus acquires a captured image, sets a plurality of parameters for generating a matching model, generates a matching model for detecting a target object based on the acquired captured image and the set plurality of parameters, and identifies an item of a common parameter for common use by a plurality of target objects and an item of an individual parameter to be determined for each of the plurality of target objects from the plurality of parameters in a case where the plurality of parameters is set and a plurality of matching models is generated for each of the plurality of target objects.

BACKGROUND Field of the Disclosure

The present disclosure relates to a technique for generating a matchingmodel.

Description of the Related Art

Techniques are known for generating a standard template corresponding toan object and a recognition template representing a difference from thestandard template and then performing template matching using thestandard template and the recognition template in object recognition, asdiscussed in Japanese Patent Application Laid-Open No. 2018-151748.

Japanese Patent Application Laid-Open No. 2018-151748 discusses a methodfor increasing the speed of matching processing in object recognitionusing template matching but does not discuss a method for adjusting anitem set for a plurality of target objects for which a model is to begenerated.

For example, in a case where the total number of a plurality of fineworkpieces or whether orientations of the workpieces are correct are tobe checked by image processing, there may be a great variety of types ofworkpieces. The workpieces may thus be similar in shape or shading toeach other but different in size from each other. In this case, it isoften difficult to prepare by generating a single matching model orseveral matching models in advance. Furthermore, generating matchingmodels corresponding to all types of workpieces in advance necessitatesa significant number of processes. Even if matching models aregenerated, it is unrealistic for an operator to select a matching modelfor each of several thousand IC chips during matching processing.

SUMMARY

Some embodiments of the present disclosure generate a suitable matchingmodel and perform matching processing each time the image processing isperformed. And some embodiments of the present disclosure relate to amethod for generating a matching model corresponding to a type of amatching target (e.g., integrated circuit (IC) chips as an example of aworkpiece in exemplary embodiments below) while reducing a load on anoperator in generating the matching model.

According to an aspect of the present disclosure, an image processingapparatus includes an acquisition unit configured to acquire a capturedimage, a setting unit configured to set a plurality of parameters forgenerating a matching model, a generation unit configured to generate amatching model for detecting a target object based on the captured imageacquired by the acquisition unit and the plurality of parameters set bythe setting unit, and an identification unit configured to identify anitem of a common parameter for common use by a plurality of targetobjects and an item of an individual parameter to be determined for eachof the plurality of target objects from the plurality of parameters in acase where the plurality of parameters is set and a plurality ofmatching models is generated for each of the plurality of targetobjects. The setting unit sets the individual parameter in generatingthe matching model. The generation unit generates the matching modelbased on the common parameter stored in advance and the individualparameter set by the setting unit.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an integrated circuit (IC) chiprepacking operation according to an aspect of the present disclosure.

FIG. 2 is flowchart illustrating an IC chip repacking operationaccording to an aspect of the present disclosure.

FIG. 3 is a diagram illustrating processing targets and workpiecesaccording to an aspect of the present disclosure.

FIG. 4 is a diagram illustrating a system configuration according to afirst exemplary embodiment.

FIGS. 5A to 5C are block diagrams illustrating hardware and softwareconfigurations according to the first exemplary embodiment.

FIG. 6 is a sequence diagram illustrating an image capturing processaccording to the first exemplary embodiment.

FIG. 7 is a flowchart illustrating a process of setting image processingsettings according to the first exemplary embodiment.

FIGS. 8A and 8B are a sequence diagram illustrating a process of imageprocessing settings according to the first exemplary embodiment.

FIG. 9 is a flowchart illustrating an inspection process using imageprocessing according to the first exemplary embodiment.

FIGS. 10A and 10B are diagrams illustrating settings for matching modelgeneration according to the first exemplary embodiment.

FIGS. 11A to 11C are diagrams illustrating settings for matching modelgeneration according to the first exemplary embodiment.

FIGS. 12A and 12B are diagrams illustrating settings for matching modelgeneration according to the first exemplary embodiment.

FIG. 13 is a flowchart illustrating a process of determining a parameterthat is to be set individually for each IC chip type according to thefirst exemplary embodiment.

FIGS. 14A to 14C are diagrams illustrating settings for matching modelgeneration according to the first exemplary embodiment.

FIG. 15 is a diagram illustrating a software configuration of anapparatus according to a second exemplary embodiment.

FIG. 16 is a flowchart illustrating a matching model generation processaccording to the second exemplary embodiment.

FIG. 17 is a flowchart illustrating a trained model data generationprocess according to the second exemplary embodiment.

FIG. 18 is a flowchart illustrating a difference calculation processaccording to the second exemplary embodiment.

FIG. 19 illustrates an example of a display of a temporal change indifference according to the second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments of the present disclosure will bedescribed below with reference to the drawings. It should be noted thateach component described in the exemplary embodiments below is a mereexample and that the scope of every embodiment is not limited to thosedescribed herein.

According to a first exemplary embodiment, integrated circuit (IC) chipsthat are a type of workpiece in production lines will be described belowas an example of a target object to be a matching target for detectionand recognition. An image processing apparatus will be described belowthat is configured to generate a matching model for a workpiece typewhile reducing a matching model generation load in a use case wherematching processing is to be performed on a large amount of workpiecesthat are similar to each other yet different from each other accordingto the present exemplary embodiment.

FIG. 1 is a diagram illustrating an operation environment to which thepresent exemplary embodiment is applied and an overview of a flow of theoperation.

FIG. 2 is a flowchart illustrating the operation. In an example of theoperation environment, a case is expected where IC chips 101 aredelivered in a state of being packed in an IC chip tray 102. After thedelivery, the illustrated operation is an operation of repacking the ICchips 101 into another IC chip tray 106 based on an order form 103. InS201, an operator A 104 of the operation brings the IC chip tray 102containing the IC chips 101. In S202, the operator A 104 then preparesthe IC chip tray 106 into which the IC chips 101 are to be repacked.

At this time, the prepared IC chip tray 106 contains no IC chips. Inother words, an empty IC chip tray is prepared. In S203, the operator A104 repacks the IC chips 101 based on the order form 103. In therepacking, how many IC chips 101 are to be repacked and how the IC chips101 are to be arranged (e.g., the number of rows and columns) aredetermined based on the order form 103. In S204, after completing therepacking of the IC chips 101 based on the order form 103, the operatorA 104 checks whether the state of the IC chips 101 repacked in the ICchip tray 106 matches the description in the order form 103. In S205, ina case where the operator A 104 after the checking determines that thestate of the IC chips 101 repacked in the IC chip tray 106 matches thedescription in the order form 103 (YES in S205), the operator A 104requests an operation leader B 105 to check the state for double-check.In S206, the operation leader B 105 checks whether the repacking of theIC chips 101 by the operator A 104 is correctly performed. In S207, in acase where the operation leader B 105 determines that the state of theIC chips 101 repacked in the IC chip tray 106 does not match thedescription in the order form 103 (NO in S207), the operation leader B105 requests the operator A 104 to correct the repacked IC chips 101 andthen recheck the corrected IC chips 101. In contrast, in a case wherethe operation leader B 105 determines that the IC chip tray 106 afterthe repacking is in a correct state (YES in S207), the processingproceeds to S208. In S208, the operation on the IC chip tray 106 iscompleted. In S209, the order form 103 is checked, and the foregoingoperation is repeated the number of times corresponding to the number ofIC chip trays that is specified in the order form 103. In a case wherethe repacking operation specified in the order form 103 is completed(YES in S209), the processing proceeds to S210. In S210, the IC chiptrays 106 in the quantity specified in the order form 103 are packaged.

An example of an operation environment has been described above, and thepresent exemplary embodiment is applied to an operation in theenvironment. The operation is specifically an operation included inareas A199 and A299 in FIGS. 1 and 2, respectively. This operation is anoperation of checking whether the number and arrangement of IC chips areas specified in an order form, and the operation of visual checking byhuman eyes as described above (area A299) is replaceable with adetermination process based on images captured by an image capturingapparatus (e.g., network camera).

FIG. 3 is a diagram illustrating types of IC chips as an imageprocessing target and types of IC chip trays for storing the IC chipsaccording to the present exemplary embodiment. The types of IC chips andIC chip trays vary over a wide range. The operation of repacking ICchips into IC chip trays is performed on various IC chips and various ICchip trays. There are also cases where different IC chips are repackedinto the same IC chip tray. In other words, determining IC chip traysuniquely does not always lead to a unique determination of IC chips.

The image processing according to the present exemplary embodiment is tocheck whether the total number and orientations of all packed IC chipsare correct. However, since there are various types of IC chips, theremay arise a situation where IC chips are similar in shape or shading toeach other but different in size from each other, and it is difficult torespond to this situation by generating a single matching model orseveral types of matching models in advance. Even if matching modelscorresponding to all IC chips can be generated in advance, thegenerating necessitates a significant number of processes, and it isunrealistic for an operator to select a matching model for each ofseveral thousand IC chips during matching processing. In other words,each time the image processing is performed, desirably a suitablematching model is generated and the matching processing is performed.

The present exemplary embodiment relates to a process for generating asuitable matching model each time the matching processing is performedand relates to a method for generating a matching model corresponding toa type of a workpiece (e.g., IC chip in the present exemplaryembodiment) while reducing a load on an operator in generating thematching model.

<System Configuration>

FIG. 4 is a diagram illustrating an entire configuration of an imageprocessing system. The system according to the present exemplaryembodiment includes apparatuses that perform the following processing.

The image processing system according to the present exemplaryembodiment is a system that captures images of articles delivered to afactory and that inspects and checks whether the number and orientationsof the articles are correct. While an example will be described belowwhere the image processing system according to the present exemplaryembodiment inspects and checks a plurality of IC chips packed in IC chiptrays, targets to be inspected and checked are not limited to IC chips.

The image processing system includes an image processing apparatus 403,an image capturing apparatus 401 (e.g., a network (NW) camera), and anoperation terminal 405. The image processing apparatus 403, the imagecapturing apparatus 401, and the operation terminal 405 communicate witheach other via a network 402. The image processing apparatus 403connects to the image capturing apparatus 401 via the network 402, andcontrols image capturing performed by the image capturing apparatus 401.

The image capturing apparatus 401 is situated to capture images of an ICchip tray 106 containing IC chips and captures images of the IC chiptray 106 under control by the image processing apparatus 403. IC chipspacked in a delivered IC chip tray 102 are arranged in the empty IC chiptray 106 based on an order form. After an operator 104 completes the ICchip arrangement, the image capturing apparatus 401 images the IC chiptray 106 and obtains captured images. The image processing apparatus 403performs matching processing on the plurality of IC chips packed in theIC chip tray 106 using a model of a single IC chip based on the capturedimages. By the foregoing process, the number and orientations of the ICchips packed in the IC chip tray 106 are inspected.

<Hardware Configuration>

FIG. 5A is a block diagram illustrating a hardware configuration of theimage processing apparatus 403 and the operation terminal 405.

The image processing apparatus 403 is configured as a computer apparatusincluding a central processing unit (CPU) 501, a read-only memory (ROM)503, a random access memory (RAM) 502, a hard disk drive (HDD) 506, aninput device 507, a display 504, an interface (I/F) 508, and a bus 509.The CPU 501 processes programs. The ROM 503 stores programs. Data forexecuting a program is loaded into the RAM 502. The HDD 506 stores data.The input device 507 and the display 504 are used in issuing controlinstructions to the programs and in registering settings information.The I/F 508 communicates with external systems.

The functions of the image processing apparatus 403 and the operationterminal 405 are realized by reading a predetermined program onhardware, such as the CPU 501 or the ROM 503 storing the programs, sothat the CPU 501 controls the units of the image processing apparatus403 and the operation terminal 405 and realizes various types ofprocessing. The hardware configuration described above and a softwareconfiguration described below are mere examples, and configurations arenot necessarily limited to those described herein according to thepresent exemplary embodiment. For example, a plurality of pieces ofhardware can cooperate together to function as a single unit, or asingle piece of hardware can function as a plurality of units.

<Software Configuration>

FIG. 5B is a block diagram illustrating a software configuration of theimage processing apparatus 403.

The software configuration of the image processing apparatus 403includes processing modules, such as a communication unit 514, a dataacquisition/storage unit 511, an image processing unit 512, and an imagecapturing apparatus control unit 513.

The communication unit 514 receives requests from the other processingmodules and performs processing, such as data transmission and receptionto and from external systems (e.g., transmission and reception ofcontrol commands to and from external devices).

The data acquisition/storage unit 511 stores data acquired throughcommunication. The data acquisition/storage unit 511 also provides andtransmits the stored data based on requests from the other processingunits. The data acquisition/storage unit 511 further sorts informationabout the stored data (e.g., data that matches a specific condition ordata that is extracted based on a specific condition) and provides thedata to the other processing units.

The image processing unit 512 performs image processing based onrequests from external systems. Results of the image processingperformed by the image processing unit 512 are stored in the dataacquisition/storage unit 511. The image processing unit 512 alsoperforms model generation processing relating to matching that is a typeof the image processing. Matching model data generated by the imageprocessing unit 512 is stored in the data acquisition/storage unit 511.

The image capturing apparatus control unit 513 performs processing to beready to communicate with the image capturing apparatus 401 connected tothe image processing apparatus 403 via the communication unit 514. Afterthe image processing apparatus 403 and the image capturing apparatus 401become communicable with each other, the image capturing apparatuscontrol unit 513 acquires settings data from the image capturingapparatus 401 or performs processing to change image capturingparameters. After setting a desired image capturing parameter to theimage capturing apparatus 401, the image capturing apparatus controlunit 513 transmits an image capturing command to the image capturingapparatus 401 via the communication unit 514, so that the imagecapturing apparatus 401 performs image capturing processing. Image datacaptured through the image capturing processing is received by the imagecapturing apparatus control unit 513 from the image capturing apparatus401 via the communication unit 514 and stored in the dataacquisition/storage unit 511 as needed.

The image processing unit 512 performs image processing using image datacaptured by the image capturing apparatus control unit 513 as input.

Details of the image processing performed by the image processing unit512 and the image capturing apparatus control unit 513 and settingprocessing for performing the image processing will be described below.

FIG. 5C is a block diagram illustrating a software configuration of theoperation terminal 405.

The operation terminal 405 transmits a request to perform desired imageprocessing to the image processing apparatus 403, receives a result ofthe request, and displays the received result in a form that can bechecked by an operator performing operations using the system. Thesoftware configuration of the operation terminal 405 includes processingmodules, such as a communication unit 523, a data acquisition/storageunit 521, and a user interface (UI) unit 522.

The communication unit 523 performs data transmission and reception toand from external systems (e.g., transmission of a command of a requestto perform image processing to the image processing apparatus 403).

The data acquisition/storage unit 521 performs data storage processing.The data acquisition/storage unit 521 also provides and transmits thestored data based on requests from the other processing units.

The data acquisition/storage unit 521 sorts information about the storeddata (e.g., data that matches a specific condition or data that isextracted based on a specific condition) and provides the data to theother processing units.

The UI unit 522 performs display processing for setting and executingthe image processing corresponding to an operation to be performed by anoperator.

Details of a process of causing the image processing apparatus 403 toperform desired image processing from the operation terminal 405 will bedescribed below.

<Sequence of Setting Process (Image Capturing Process)>

FIG. 6 is a sequence diagram illustrating an image capturing processaccording to the present exemplary embodiment.

An area A691 corresponds to a process of a connection between theoperation terminal 405 and the image processing apparatus 403.

The operation terminal 405 and the image processing apparatus 403 needto be recognizable to each other and connected to each other in orderfor the operation terminal 405 to transmit various processing requeststo the image processing apparatus 403 and for the image processingapparatus 403 to transmit processing results to the operation terminal405 in response to the requests.

In S601, the operation terminal 405 transmits a connection request tothe image processing apparatus 403. In S602, the image processingapparatus 403 establishes a connection with the operation terminal 405.

For example, the operation terminal 405 and the image processingapparatus 403 connect to each other via Hypertext Transport Protocol(HTTP) communication and establish a connection. A method using HTTPcommunication herein is not limited.

An area A692 corresponds to a process of establishing a connection withthe image capturing apparatus 401 controlled by the image processingapparatus 403.

In S603, the image processing apparatus 403 performs polling on thenetwork 402 to check repeatedly whether the image capturing apparatus401 is connected to the network 402.

In S604, a connection of the image capturing apparatus 401 to thenetwork 402 is detected.

In S605, the image processing apparatus 403 transmits a connectionrequest to the image capturing apparatus 401. In S606, the imagecapturing apparatus 401 establishes a connection with the imageprocessing apparatus 403 and transmits a connection completionnotification.

An area A693 corresponds to a process of adjusting and setting an imagecapturing condition for the image capturing apparatus 401.

In S607 and S608, the operation terminal 405 transmits an imagecapturing parameter change request to the image capturing apparatus 401via the image processing apparatus 403. In S609, the image capturingapparatus 401 performs processing to change an image capturing parameterbased on the received image capturing parameter change request. In acase where the image capturing parameter change request is abnormal(e.g., a value to which the parameter is requested to be changed isoutside a range), an error is returned. In S610 and S611, after normallycompleting the parameter changing processing based on the received imagecapturing parameter change request, the image capturing apparatus 401transmits a change completion notification together with the changedparameter value to the operation terminal 405 via the image processingapparatus 403.

An area 694 corresponds to a process of image capturing by the imagecapturing apparatus 401.

After the adjustment of the image capturing condition is completed as aresult of the process of adjusting and setting the image capturingcondition for the image capturing apparatus 401 in S607 to S611, in S612and S613, the operation terminal 405 transmits an image capturingrequest to the image capturing apparatus 401 via the image processingapparatus 403. In S614, the image capturing apparatus 401 receives theimage capturing request from the image processing apparatus 403 andperforms image capturing. In S615 and S616, the image capturingapparatus 401 transmits captured data to the operation terminal 405 viathe image processing apparatus 403.

<Flowchart of Setting Process of Image Processing>

FIG. 7 is a flowchart illustrating a setting process of image processingaccording to the present exemplary embodiment.

In S701, the image processing apparatus 403 sets a parameter forgenerating matching models based on a request from the operationterminal 405. In S702, the image processing apparatus 403 generates amatching model based on the request from the operation terminal 405.Details of the matching model generation processing in S701 and S702will be described below. In S704 (YES in S703), the image processingapparatus 403 sets a parameter for performing matching processing basedon the request from the operation terminal 405 using the matching modelgenerated in S702 described above. Since this is the first time toperform the setting processing, it is determined in S703 that theparameter for matching processing is to be adjusted (YES in S703). In acase where a matching result is not a desired result as described below,whether to re-adjust the parameter for matching processing (whether YESor NO in S703) is determined by a user based on the matching result. InS705, the image processing apparatus 403 performs matching processingbased on the request from the operation terminal 405. Details of thematching processing performed in S704 and S705 will be described below.In a case where the performance result is a desired result (YES inS706), the processing proceeds to S708. In S708, the image processingapparatus 403 stores various types of information in the dataacquisition/storage unit 511 based on the request from the operationterminal 405. Specifically, the parameter for generating matching models(that is set in S701), the parameter for performing matching processing(that is set in S704), and the matching model (that is generated inS702) are stored.

In a case where the performance result is not a desired result (NO inS706) and where it is determined that the process is to be performedagain from the matching model generation in S707 (YES in S707), theprocessing returns to S701 to be performed again. After a matching modelis generated in S702, and in a case where it is determined that theparameter for matching processing is to be set again (YES in S703), S704is performed again. In a case where the parameter for matchingprocessing is to be used without adjustment (NO in S703), the processingproceeds to S705. In S705, the matching processing is performed usingthe new generated matching model.

Further, in a case where the performance result is not a desired result(NO in S706) and where the matching model generation is not to beperformed (NO in S707), the processing proceeds to S704. In S704, theparameter for performing matching processing is set and adjusted again.The subsequent process is as described above, and the matching modelgeneration and adjustment and the matching processing performance andadjustment are repeated until a desired matching performance result isobtained.

<Sequence of Matching Model Generation and Matching Processing>

The process of generating a matching model (S701, S702) and the processof performing matching processing (S704, S705) described above withreference to FIG. 7 will be described in more detail below withreference to a sequence diagram illustrated in FIGS. 8A and 8B.

The process of generating a matching model will now be described.

In S801 to S806, the operation terminal 405 first causes a connectedimage capturing apparatus 401 to perform image capturing. Details of theimage capturing are as described above (S612 to S616).

In S807, the operation terminal 405 sets the parameter for generatingmatching models that is to be applied to the acquired captured imagedata to generate a matching model. Each time the parameter forgenerating matching models is set, a state of a generated matching modelis displayed for checking on the operation terminal 405, and theparameter setting is performed to generate a desired matching modelwhile an adjustment level is checked.

In S808, after the adjustment of the parameter for generating matchingmodels is completed, the operation terminal 405 transmits a matchingmodel generation request to generate a matching model to the imageprocessing apparatus 403.

In S809, the image processing apparatus 403 receives the matching modelgeneration request from the operation terminal 405 and acquires thecaptured image data stored in S805. In S810, the image processingapparatus 403 generates a matching model. In S811, the image processingapparatus 403 transmits a matching model generation completionnotification to the operation terminal 405. In S812, the operationterminal 405 acquires the generated matching model based on the matchingmodel generation completion notification data transmitted from the imageprocessing apparatus 403 and displays the acquired matching model in acheckable form. At this time, in S813 to S816, the image processingapparatus 403 stores the generated matching model data and the parameterdata for generating matching models in the data acquisition/storage unit511.

The process of performing matching will now be described.

In S821 to S826, the operation terminal 405 first transmits an imagecapturing request to the connected image capturing apparatus 401.Details of the image capturing are as described above (S612 to S616).

In S827, the operation terminal 405 sets a parameter to be applied tothe acquired captured image data for performing matching processing. InS828, in order to set a parameter for performing matching processing andto check a matching level, the operator 104 transmits a request toperform matching processing to the image processing apparatus 403 usingthe operation terminal 405.

In S829, the image processing apparatus 403 receives the request toperform matching processing from the operation terminal 405 and acquiresthe captured image data stored in S825. In S830, the image processingapparatus 403 performs matching processing. In S831, the imageprocessing apparatus 403 transmits matching processing performanceresult data to the operation terminal 405.

In S832, the operation terminal 405 performs display processing forchecking the matching processing performance result generated based onthe matching processing performance result data transmitted from theimage processing apparatus 403. In S833 and S834, the image processingapparatus 403 stores the matching processing performance result data andthe setting parameter for performing matching in the dataacquisition/storage unit 511.

<Flowchart of Inspection Process Using Image Processing>

FIG. 9 is a flowchart illustrating a process of performing imageprocessing according to the present exemplary embodiment.

The image processing apparatus 403 starts performing image processingand matching based on a request received from the operation terminal405.

In S901, the image processing apparatus 403 acquires the parameter datafor generating matching models and the setting condition for performingmatching that are stored in S708 from the data acquisition/storage unit511 to perform matching processing.

In S902, the image processing apparatus 403 performs image capturingprocessing for generating a matching model.

In S903, after acquiring image data output through the image capturingprocessing in S902 from the image capturing apparatus 401, the imageprocessing apparatus 403 generates a matching model based on theacquired parameter data for generating matching models.

The matching processing is then performed using the matching modelgenerated in S903. Specifically, the image processing apparatus 403performs, in S904, image capturing processing on a matching target. InS905, after acquiring image data output through the image capturingprocessing in S904 from the image capturing apparatus 401, the imageprocessing apparatus 403 performs matching processing based on theacquired setting condition parameter data for performing matching.

Lastly, in S906, the image processing apparatus 403 stores the performedmatching processing result in the data acquisition/storage unit 511.

<Setting Item UI for Generating Matching Models>

A specific example of setting items for generating matching models(S903) will now be described.

First, an area setting is performed for designating a target for which amatching model is to be generated. As illustrated in FIG. 10A, an areatype 1001 for designating a shape for setting an area is set, andcoordinates 1002 to 1005 of four points are set in a case where the areatype 1001 is, for example, rectangle. The coordinates 1002 to 1005 offour points can also be set by operating or changing a rectangle 1006 ona UI.

As illustrated in FIG. 10B, there are other parameters 1011 to 1017 forgenerating matching models. The other parameters 1011 to 1017 include,for example, parameters for setting an upper limit 1014 and a lowerlimit 1015 of contrast for detection. The operator 104 sets the settingvalues within minimum and maximum values using the UI and generatesmatching models.

Thereafter, when a GENERATE MODEL button 1007 is pressed, the imageprocessing apparatus 403 generates a matching model as in S801 to 5817described above. At this time, the processing is performed such that aname and identification (ID) of the generated matching model are storedas information for identifying the generated matching model thereafterin association with time and date of the generation, generated images,and the parameters for the generation. Whether the matching modelgenerated as described above is appropriate is inspected through theprocessing described above with reference to FIG. 7.

An area setting for generating a matching model will now be describedwith reference to FIG. 11A.

In a case where an IC chip 1101 illustrated in FIG. 11A is to be set asa target for which a matching model is to be generated, an area is setas an area (x1, y1), (x1+x2, y1+y2). At this time, the coordinates ofthe area are coordinates with respect to an operation table 406, and theposition of an IC chip tray 1102 is arranged and fixed in contact withan IC chip tray fixing device 407 by the operator 104. Thus, coordinatesfor generating a matching model for the IC chip 1101 stored in the ICchip tray 1102 are set to the area (x1, y1), (x1+x2, y1+y2).

A matching model is thereby generated using the set area. An area 1110illustrated in FIG. 11B is then designated as a matching processingtarget range using the matching model, and the matching processing isperformed. As a result, the number of detected targets is nine asillustrated by IC chips 1111 to 1119. Furthermore, if information forrecognizing the top, bottom, right, left of each IC chip is available,orientations are also checkable.

In a case where a matching model is to be generated for IC chipsillustrated in FIG. 11C, an area (x3, y3), (x3+x4, y3+y4) is set.

This setting is different from the IC chip area (x1, y1), (x1+x2, y1+y2)illustrated in FIG. 11A. As described above, the parameters forgenerating matching models (e.g., the upper and lower limit values ofcontrast) can be the same for the IC chips illustrated in FIGS. 11A and11C. However, different areas for which a matching model is to begenerated need to be set for different types of IC chips.

Thus, the parameters to be set in generating a matching model aredivided into items of parameters that can be used for different IC chipsand items of parameters that cannot, and the former and the latter areset differently. Specifically, as illustrated in FIG. 12A, check statesof checkboxes 1201 to 1206 are configured to be changeable in generatinga matching model. A check is entered into each of the checkboxes 1201 to1206 that corresponds to a parameter (common parameter) to be used fordifferent IC chips. In contrast, no check is entered into the checkboxes1201 to 1206 that correspond to a parameter (individual parameter) to beused for different IC chips (i.e., a parameter for which differentvalues are to be set for different IC chips in generating a matchingmodel). Information about whether a check is entered is stored asparameter data for generating matching models.

A check is entered in the checkbox 1201 in FIG. 12A and checks areentered in checkboxes 1211 to 1217 in FIG. 12B, so that the itemscorresponding to the checkboxes 1201 and 1211 to 1217 are stored ascommon parameters for common use. The other parameters are stored asindividual parameters to be set individually for each IC chip type.Since the common parameters for common use are used in generatingmatching models for a plurality of types of IC chips, parameter valuesthat are set are also stored.

As described above, according to the present exemplary embodiment, in acase where there are a large variety of types of IC chips for which amatching model is to be generated, the setting of whether a parameter isan item of a parameter to be set individually for each IC chip type isset by an operation on the UI and the set setting is stored by anoperation on the UI. A process of setting a parameter as a parameter tobe set individually for each IC chip type or as a parameter not to beset individually for each IC chip type by an operation on the UI asdescribed above is performed in S701 in FIG. 7 described above.

A method for setting an item of a parameter to be set individually foreach IC chip type and an item of a parameter for common use via the UIas described above with reference to FIGS. 12A and 12B in generating amatching model will be described with reference to FIG. 13.

S701 to S707 of generating a matching model are similar to thoseillustrated in FIG. 7. In S701 to S707 and S1301, the process from thematching model generation to the performance of the matching processingfor checking the result is repeated to check a plurality of types ofcombinations of an IC chip for which a matching model is to be generatedand an IC chip tray for the IC chip (YES in S1302). During the process,in S1301, a history of adjusted setting parameters for generatingmatching models is stored. After the repeated process of performance,adjustment, and checking is completed (NO in S1302), the processingproceeds to S1303. In S1303, the image processing apparatus 403 refersto the history of setting parameters for generating matching models thatis stored in S1301 and checks a history of changes (i.e., changes invalue of each parameter).

Specifically, the parameters in the history are divided into parameterswith an amount of change having reached a preset threshold value (i.e.,parameters with a great amount of change) and parameters with an amountof change not having reached the present threshold value (i.e.,parameters with a small amount of change or with no change), and theresults are reflected to the presence/absence of a check in theparameter settings in FIGS. 12A and 12B described above. This enablesthe user generating matching models to recognize the change levels ofthe parameters for generating matching models for a plurality of ICchips for which a matching model is to be generated. Specifically,recommended values for the parameters to be set individually andrecommended values for the parameters not to be set individually aredetermined for each IC chip type in generating a matching model in thecases to which the present exemplary embodiment is applied. Finally theuser can check values of the checkboxes on the UI and determine a valueby entering or clearing a check by a user operation.

Lastly, in S708, the image processing apparatus 403 stores the settingvalues determined in S1303 in the data acquisition/storage unit 511.

As described above, in order to perform matching processing on aplurality of types of IC chips and to check the number and orientationsof the IC chips, it is desirable to generate a matching model each timefor each IC chip of a different type and to perform matching using thegenerated matching model. Furthermore, the process is described above ofidentifying whether the parameters are parameters to be set individuallyfor each of different types of IC chips for which a matching model is tobe generated and of storing the information.

A method by which a parameter to be set individually for each IC chiptype that is determined and stored in S1303 and S708 described above isset during the matching processing in an actual inspection process willbe described below with reference to FIG. 14A.

A flow of the inspection process will now be described with reference toFIG. 9 described above. A process of setting a parameter to be setindividually for each IC chip type described below is performed in S903in FIG. 9.

In FIG. 9, the image processing apparatus 403 starts matching processingbased on a request from the operation terminal 405, and the matchingmodel generation processing is performed after execution in S901 andS902. The image processing apparatus 403 acquires parameter data forgenerating matching models from the data acquisition/storage unit 511.The image processing apparatus 403 divides the parameters for generatingmatching models into parameters to be set individually for each type ofIC chips for which a matching model is to be generated and parametersnot to be set individually for each IC chip type. The image processingapparatus 403 transmits the data to the operation terminal 405. Theoperation terminal 405 displays a setting screen (FIG. 14A) for theparameters received from the image processing apparatus 403, theparameters for which a matching model is to be generated and theparameters to be set individually for each IC chip type among theparameters for which a matching model is to be generated. Specifically,for the parameters to be set individually for each IC chip type, theuser can change and move coordinates by operating an area 1401 displayedon the UI using a cursor 1402. A UI via which values 1403 to 1406 of thearea coordinates are settable is also displayed, and the user can setthe values 1403 to 1406. The area 1401 and the values 1403 to 1406 ofthe area coordinates are displayed such that the UI coordinates and thevalues always correspond.

Meanwhile, the parameters that are not to be set individually for eachIC chip type and for which a matching model is to be generated aredisplayed such that the parameters are unchangeable (item 1407).

As described above, the user can perform an operation to set theparameters that are to be set individually for each IC chip type and forwhich a matching model is to be generated in the inspection processthrough the processing by the image processing apparatus 403 and theoperation terminal 405.

Initial setting values of the displayed parameters in FIG. 14A that areto be set individually for each IC chip type and for which a matchingmodel is to be generated will now be described. In FIG. 14A describedabove, preset fixed values are expected to be used as initial settingvalues. In this case, however, the user operation is always needed toset the parameters that are to be set individually for each IC chip typeand for which a matching model is to be generated to a desired value.

In another example, specific image processing (e.g., image processing todetect an area corresponding to contrast values in a preset range) isperformed and initial setting values are determined based on thedetection result before the matching model generation processing in S903in FIG. 9. Specifically, in FIG. 14B, coordinates of an area 1411 havingthe smallest x- and y-coordinates among the detected IC chips 1411 to1419 are acquired, and the acquired coordinates are used as initialsetting values of the area 1401 of FIG. 14A. Thus, an area close to anarea of IC chips for which a matching model is to be generated is set asinitial setting values (recommended values), and an area adjustmentoperation performed thereafter by the user can be minimized.

For the displayed parameters in FIG. 14A that are to be set individuallyfor each IC chip type and for which a matching model is to be generatedas described above, a plurality of displays of initial setting valuescan be implemented. A plurality of areas, such as areas 1421 and 1422 inFIG. 14C, can also be displayed. In this case, if the user selects oneof the areas 1421 and 1422 (e.g., if the user starts an operation tofirst change or move the coordinates of the area 1422 using the cursor1402), the display of the other area 1421 can be hidden.

As described above, the user sets the parameters to be set individuallyfor each IC chip and for which a matching model is to be generated inthe inspection process through the processing by the image processingapparatus 403 and the operation terminal 405, and the values are set bypressing a SET button 1408. A matching model is generated using the setarea, and S904 and subsequent operations in the flowchart in FIG. 9 areperformed, and the matching processing is performed.

As a result, an image processing apparatus configured to generate amatching model for a workpiece type is realized while reducing amatching model generation load in a use case where matching processingis to be performed on a large amount of workpieces that are similar toeach other yet different from each other according to the presentexemplary embodiment.

As described above in the first exemplary embodiment, the parameters tobe set individually for each IC chip type are set in S903 of generatinga matching model each time a single workpiece (IC chip) is inspected inthe inspection process illustrated in FIG. 9. Specifically, the userdetermines the parameters by selecting and operating a UI displayed onthe operation terminal 405 based on values output from the imageprocessing apparatus 403.

The user selection and operation are desirably a minimum operation. Amethod for minimizing the user selection and operation in setting theparameters to be set individually for each IC chip type according to thepresent exemplary embodiment will be described below.

FIG. 15 is a block diagram illustrating a software configuration of theimage processing apparatus 403 according to a second exemplaryembodiment.

The software configuration of the image processing apparatus 403includes processing modules, such as the communication unit 514, thedata acquisition/storage unit 511, the image processing unit 512, theimage capturing apparatus control unit 513, a training unit 1501, and aninference unit 1502.

Operations of the communication unit 514, the data acquisition/storageunit 511, the image processing unit 512, and the image capturingapparatus control unit 513 are similar to those according to the firstexemplary embodiment.

The training unit 1501 generates trained model data for inferring anarea of IC chips for which a matching model is to be generated based ondata for training that consists of images and information about an areaof IC chips for which a matching model is to be generated.

The inference unit 1502 acquires captured image data and determines anarea of IC chips for which a matching model is to be generated asdescribed below.

A process of accumulating determined area coordinate values forgenerating matching models as data for training, learning the values,and applying an inferred value to a recommended value for areacoordinates for generating a matching model will be described below withreference to FIG. 16. In S901, the image processing apparatus 403acquires the setting values for generating matching models and forperforming matching that are adjusted by the user in the setting processbased on a request from the operation terminal 405 in the inspectionprocess according to the present disclosure. In S902, the imageprocessing apparatus 403 performs image capturing processing forgenerating a matching model. At this time, in S1601, captured image datafor generating a matching model is stored and accumulated in the dataacquisition/storage unit 511 (image 1613). The stored and accumulateddata is used as data for training by the training unit 1501 of the imageprocessing apparatus 403.

In S903, the image processing apparatus 403 performs matching modelgeneration processing. S1602 to S1608 are performed during the matchingmodel generation processing in S903.

Specifically, a trained model 1614 for performing the inference that isperformed by the inference unit 1502 of the image processing apparatus403 is acquired. In S1603, the inference unit 1502 of the imageprocessing apparatus 403 performs inference on the image data capturedin S902 using the acquired trained model 1614. In S1604, the imageprocessing apparatus 403 applies a value output through the performedinference as a recommended value for area coordinates for generating amatching model and transmits the recommended value to the operationterminal 405, and the operation terminal 405 displays the recommendedvalue for area coordinate on the UI. In a case where there is no trainedmodel 1614, the inference is not performed, and the initial settingvalues described above with reference to FIGS. 14A, 14B, and 14Caccording to the first exemplary embodiment are applied to recommendedvalues for area coordinates for a matching model that are displayed inS1604. In S1605, the recommended values for the area coordinates for amatching model that are output from the image processing apparatus 403are stored in the data acquisition/storage unit 511 (image 1611). Asdescribed below, the recommended values are stored for use by the imageprocessing apparatus 403 in calculating a difference between therecommended values and the area coordinates for generating a matchingmodel that are finally determined by the user.

In S1606, the user checks the recommended values for the areacoordinates for a matching model that are displayed in S1604 and theimages captured in S902 and adjusts the area for generating a matchingmodel. In S1607, after completing the adjustment operation in S1606, theuser determines an area setting for generating a matching model. InS1608, the image processing apparatus 403 stores and accumulates thedetermined coordinate values of the area for generating a matching modelin the data acquisition/storage unit 511 (image 1612). At this time, thedetermined values of the area coordinates for generating a matchingmodel that are stored and accumulated are stored in association with therecommended values. As described below, the determined values are storedfor use by the image processing apparatus 403 in calculating adifference between the recommended values and the area coordinates forgenerating a matching model that are finally determined by the user.

Subsequent S904 to S906 are similar to those in the inspection processdescribed above with reference to FIG. 9 according to the firstexemplary embodiment.

FIG. 17 is a flowchart illustrating a process of generating trainedmodel data that is performed by the training unit 1501.

In S1701, the image processing apparatus 403 acquires the determinedvalues of the area coordinate for generating a matching model that arestored in S1607 and the captured image data stored in S1613 from thedata acquisition/storage unit 511. A target for the determined values ofthe area coordinates for generating a matching model that are acquiredherein is data (newest unprocessed data since the previous time) that isnot a target in previous training processing by the training unit 1501of the image processing apparatus 403.

In S1702, the training unit 1501 of the image processing apparatus 403performs training processing using the data acquired in S1701 to reflectthe acquired data to the previously-generated trained model data.

In S1703, the training unit 1501 of the image processing apparatus 403generates, after completing the training processing, trained model dataand stores the generated trained model data in the dataacquisition/storage unit 511 (trained model 1614). At this time, thetrained model data is stored in a form that is uniquely identifiable(the data is named based on an operation flow type, time/dateinformation, and/or content of training target data). In S1704, thetraining unit 1501 of the image processing apparatus 403 notifies, aftercompleting the storage of the trained model data, the operation terminal405 that the generation and storage of the trained model data iscompleted. From the trained model data generation and storage completionnotification to the operation terminal 405, the user recognizes that theinference described below can be performed using new trained model data.

Since it takes some time to complete the trained model data generationprocessing, the processing can be started automatically at fixed timeafter daily operation ends, or the trained model data generationprocessing can be started based on a start instruction from the user.

FIG. 18 is a flowchart illustrating a process of calculating adifference between recommended values output through the inference anddisplayed by the inference unit 1502 of the image processing apparatus403 and area coordinates for generating a matching model that arefinally determined by the user.

In S1801, the image processing apparatus 403 acquires the areacoordinate values 1612 stored in S1608 and the recommended values 1611for the area coordinates for generating a matching model that are storedin association with the area coordinate values 1612 from the dataacquisition/storage unit 511. In S1802, the image processing apparatus403 calculates the difference between the acquired data. The calculateddifference value indicates an adjustment level of adjustment made by theuser from the recommended values. In S1803, the image processingapparatus 403 transmits the calculated difference value data to theoperation terminal 405, and the operation terminal 405 displays temporalchanges in the adjustment level of adjustment made by the user based onthe difference value data and the information associated with thedifference value data.

A display example of S1803 is illustrated in FIG. 19.

A checkbox 1902 is a UI for selecting whether to apply the calculatedrecommended values automatically (without checking and determiningoperations by the user) to area coordinates for generating a matchingmodel.

In a case where a check is entered in the checkbox 1902, the useroperations of adjusting and determining area coordinates for generatingmatching models according to the first and second exemplary embodimentsare omitted. Specifically, in a case where an adjustment made by theuser on the UI from an area coordinate for generating a matching modelthat is specified as a recommended value is within an allowable range(less than or equal to a threshold value) as illustrated in FIG. 19, therecommended value is used as a determined value without adjustment, andthe matching model generation processing is performed.

FIG. 19 illustrates an example where an adjustment made by the user froman area coordinate specified as a recommended value on the UI is withinthe allowable range (less than or equal to the threshold value) (e.g.,the difference between the recommended value output through theinference and the determined value of the area coordinate that isfinally determined by the user converges over time). However, there maybe a case where the foregoing result is not obtained and the differencedoes not converge but rather diverges over time.

This case is against the reduction of the amount of user operation thatis an object of the present exemplary embodiment. Thus, the user is tobe notified that a method according to the present exemplary embodimentis not acting effectively.

Specifically, the image processing apparatus 403 detects a possibilitythat a temporal change 1901 illustrated in FIG. 19 will not reach anexpected adjustment amount within a predetermined time period andnotifies the operation terminal 405 about the possibility so that thisis displayed on the UI. The predetermined time period and the expectedadjustment amount can be preset by the user, or the image processingapparatus 403 can include a unit including a convergence predictionprogram.

As described above, the second exemplary embodiment minimizes the userselection and operation in setting a parameter that is to be setindividually for each IC chip type.

Some embodiments are also realized by performing the followingprocessing. Specifically, software (program) for realizing the functionsof the above-described exemplary embodiments is supplied to a system oran apparatus via a network or various storage mediums, and a computer(or a CPU or a micro-processing unit (MPU)) of the system or theapparatus reads the program and executes the read program.

OTHER EMBODIMENTS

Some embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executescomputer-executable instructions (e.g., one or more programs) recordedon a storage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer-executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer-executable instructions. The computer-executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has described exemplary embodiments, it isto be understood that some embodiments are not limited to the disclosedexemplary embodiments. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims priority to Japanese Patent Application No.2021-065654, which was filed on Apr. 8, 2021 and which is herebyincorporated by reference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising at leastone processor configured to function as the following units: anacquisition unit configured to acquire a captured image; a setting unitconfigured to set a plurality of parameters for generating a matchingmodel; a generation unit configured to generate a matching model fordetecting a target object based on the captured image acquired by theacquisition unit and the plurality of parameters set by the settingunit; and an identification unit configured to identify an item of acommon parameter for common use by a plurality of target objects and anitem of an individual parameter to be determined for each of theplurality of target objects from the plurality of parameters in a casewhere the plurality of parameters is set and a plurality of matchingmodels is generated for each of the plurality of target objects, whereinthe setting unit sets the individual parameter in generating thematching model, and wherein the generation unit generates the matchingmodel based on the common parameter stored in advance and the individualparameter set by the setting unit.
 2. The image processing apparatusaccording to claim 1, wherein the identification unit changes the itemof the common parameter and the item of the individual parameter.
 3. Theimage processing apparatus according to claim 1, wherein the settingunit sets a value of the individual parameter to an initial settingvalue and changes the value of the individual parameter based on a useroperation.
 4. The image processing apparatus according to claim 1,wherein the setting unit sets a value of the individual parameter basedon a result of image processing on the captured image in generating thematching model.
 5. The image processing apparatus according to claim 1,wherein the setting unit changes at least one value of the plurality ofparameters based on a user operation, and wherein the identificationunit identifies an item as the individual parameter based on a changehistory of at least one value of the plurality of parameters based onthe user operation.
 6. The image processing apparatus according to claim1, wherein the setting unit changes at least one value of the pluralityof parameters based on a user operation, and wherein the identificationunit identifies an item as the individual parameter based on a changeamount of at least one value of the plurality of parameters based on theuser operation.
 7. A method for controlling an image processingapparatus, the method comprising: acquiring a captured image; setting aplurality of parameters for generating a matching model; generating amatching model for detecting a target object based on the acquiredcaptured image and the set plurality of parameters; and identifying anitem of a common parameter for common use by a plurality of targetobjects and an item of an individual parameter to be determined for eachof the plurality of target objects from the plurality of parameters in acase where the plurality of parameters is set and a plurality ofmatching models is generated for each of the plurality of targetobjects, wherein the individual parameter is set in generating thematching model, and wherein the matching model is generated based on thecommon parameter stored in advance and the set individual parameter. 8.A non-transitory computer-readable storage medium storing a program forcausing a computer to perform operations comprising: acquiring acaptured image; setting a plurality of parameters for generating amatching model; generating a matching model for detecting a targetobject based on the acquired captured image and the set plurality ofparameters; and identifying an item of a common parameter for common useby a plurality of target objects and an item of an individual parameterto be determined for each of the plurality of target objects from theplurality of parameters in a case where the plurality of parameters isset and a plurality of matching models is generated for each of theplurality of target objects, wherein the individual parameter is set ingenerating the matching model, and wherein the matching model isgenerated based on the common parameter stored in advance and the setindividual parameter.